On the Estimation of the Expected Performance of a Metaheuristic on a Class of Instances

This paper discusses the problem of estimating, on the basis of a given number of say N experiments, the expected performance of a metaheuristic on a class I of benchmark problem instances. The problem of the empirical estimation of the expected behavior of a stochastic optimization algorithm has great relevance both in academic studies and in practical applications. This is particularly true for metaheuristics, a class of stochastic optimization algorithms for which gaining an analytical insight appears rather problematic. In the paper, the estimation problem is formally posed in a probabilistic framework and the main theorems on the empirical estimation of the expected performance of a metaheuristic are enunciated and proved. In particular, the paper proves that the widely adopted methodology consisting in considering K instances and running the metaheuristic n times on each (with K × n = N) is a suboptimal choice. Indeed, contrary to popular belief, one single run on each of N independently selected instances provides the most reliable estimation.